Skip to content

feat(cli-vector): New cli to create and join mbtiles for vector map. BM-1268 #3435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 31 commits into from
Jun 4, 2025

Conversation

Wentao-Kuang
Copy link
Contributor

@Wentao-Kuang Wentao-Kuang commented Apr 28, 2025

Motivation

We want to move the Vector ETL work from basemaps-team into basemaps/packges/cli-vector.

The new argo workflow for the Vector shortbread ETL, Need at least 3 cli to make it work.
image

  • extract cli that load the schemas and source data, then compare with cache mbtiles in the s3 cache bucket to output updated layers(layer version update, new configurations updates in schema, or new cli version in packages) that need to process.
  • create cli that load stac item from extract cli to create individual mbtiles, then push into s3 cache bucket.
  • join cli will load all the cached mbtiles to join into the output mbtiles

Modifications

  • This PR tidy up all the 3 clis and put into a test container to run tests in argo.
  • update the cli container to include tippecanoe.
  • update the cli container to node 24, in order to use node:sqlite
  • update the @types/node to 22, in order to use node:sqlite

Verification

@Wentao-Kuang Wentao-Kuang changed the title feat(cli-vector): New cli to create mbtiles for vector map. BM1268 feat(cli-vector): New cli to create mbtiles for vector map. BM-1268 Apr 28, 2025
tawera-manaena and others added 5 commits May 26, 2025 10:43
… properties BM-1268 (#3440)

1. This pull request extends the [feat/etl-create] branch, not the
[master] branch.
2. The changed files contain short-circuiting code to enable faster
debugging that needs to be purged before approval.

---

### Motivation

We are currently migrating our Vector ETL system into basemaps as a new
`cli-vector` package. We are also re-building the system in the process.
As a part of the re-build, we have streamlined the system's architecture
in the following four steps:

1. `extract`
2. `create`
3. `join`
4. `analyse`

This work implements an extension to the base branch's implementation of
the `create` CLI command. This work introduces a series of functions for
handling dataset features of specific Shortbread layers for which we
require special tagging.

### Modifications

| Diagram |
| - |
| ![][diagram1] |

1. Implements a switch-style pattern for identifying the Shortbread
layer to which a feature is assigned.
2. Implements bespoke functions for overriding a feature's metadata and
properties for the following Shortbread layers:
   - `contours`
- `place_labels` - This one is weird. We will need to discuss the
approach for this one.
- `pois` - This one doesn't actually do anything as it never receives
any features that will meet its conditions for special tagging. We can
discuss this also.
   - `public_transport`
- `street_labels` - This one's logic is very similar to the `streets`
logic.
- `streets` - This one's logic is very similar to the `street_labels`
logic.
   - `water_polygons`

#### Processing a Vector Stac Item

The system processes a Stac Item file created via the **cli-vector**
package's new `extract` command like so:

1. Parse the file as a runtime `VectorStacItem` type.
2. Download the source file (i.e. a GeoPackage file).
3. Convert the source file into an NDJSON file.
4. Parse each line (i.e. feature) of the NDJSON file, generalise it
(i.e. simplify, add/remove attributes, or discard).
5. Process the collection of generalised features into an mbtiles file.
6. Upload a copy of the mbtiles file to the same directory as the Stac
Item file.
7. Update the Stac Item file's contents and overwrite it.

| Diagram |
| - |
| ![][diagram2] |

### Verification

1. Established test cases for the special tagging functions for the
following Shortbread layers:
   - `contours`

More to come if desired.

<!-- external links -->

[feat/etl-create]: https://github.com/linz/basemaps/tree/feat/etl-create
[master]: https://github.com/linz/basemaps/tree/master

[diagram1]:
https://github.com/user-attachments/assets/6a405417-59fb-4127-8630-cb0901c12618
[diagram2]:
https://github.com/user-attachments/assets/2994bd5d-c668-4b99-957e-57623da7c6a1

---------

Co-authored-by: Wentao Kuang <[email protected]>
@Wentao-Kuang Wentao-Kuang added the container build pull request container for testing label May 25, 2025
@Wentao-Kuang Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels May 25, 2025
@Wentao-Kuang Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels May 26, 2025
@Wentao-Kuang Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels May 26, 2025
@Wentao-Kuang Wentao-Kuang removed the container build pull request container for testing label May 26, 2025
@Wentao-Kuang Wentao-Kuang added the container build pull request container for testing label Jun 3, 2025
@Wentao-Kuang Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jun 3, 2025
@Wentao-Kuang Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jun 3, 2025
@Wentao-Kuang Wentao-Kuang added container build pull request container for testing and removed container build pull request container for testing labels Jun 3, 2025
ccbblin
ccbblin previously approved these changes Jun 3, 2025
@Wentao-Kuang Wentao-Kuang changed the title feat(cli-vector): New cli to create mbtiles for vector map. BM-1268 feat(cli-vector): New cli to create and join mbtiles for vector map. BM-1268 Jun 4, 2025
@Wentao-Kuang Wentao-Kuang enabled auto-merge June 4, 2025 02:03
@Wentao-Kuang Wentao-Kuang added this pull request to the merge queue Jun 4, 2025
Merged via the queue into master with commit 8cbef0b Jun 4, 2025
14 of 15 checks passed
@Wentao-Kuang Wentao-Kuang deleted the feat/etl-create branch June 4, 2025 02:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
container build pull request container for testing
Development

Successfully merging this pull request may close these issues.

4 participants